Papers with subword tokenization
BERT is Not an Interlingua and the Bias of Tokenization (D19-61)
Copied to clipboard
| Challenge: | Cananical Correlation Analysis (CCA) of the internal representations of a pre- trained, multilingual BERT model reveals that the model partitions representations for each language rather than using a common, shared, interlingual space. |
| Approach: | They propose to use a multilingual BERT model to partition representations for each language rather than using a common, shared, interlingual space. |
| Outcome: | The results show that the model partitions representations for each language rather than using a common, shared, interlingual space. |
CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Currently, there is no dataset containing compound and non-compound words across languages . however, current LLMs perform poorly on words tokenized unfavorably by subword tokenization. |
| Approach: | They propose to use a Wiktionary dataset to evaluate large language models on decompounding . they find that current LLMs perform poorly on words tokenized unfavorably . |
| Outcome: | The proposed model outperforms the best unsupervised models by 13.9% accuracy on average. |
Towards the Machine Translation of Scientific Neologisms (2025.coling-main)
Copied to clipboard
| Challenge: | Scientific research continually discovers and invents new concepts, which are then referred to by new terms, neologisms, or nenonyms. |
| Approach: | They propose to leverage term definitions to translate neologisms with Large Language Models . they find that LLMs generate terms from co-hyponyms and terms sharing the same derivation paradigm . |
| Outcome: | The proposed model can generate terms from co-hyponyms and terms sharing the same derivation paradigm. |
Subword Pooling Makes a Difference (2021.eacl-main)
Copied to clipboard
| Challenge: | Contextual word-representations use subword tokenization to handle large vocabularies and unknown words. |
| Approach: | They propose to use the first subword for morphological probing, POS tagging and NER to pool multiple subwords that correspond to a single word in contextual language models. |
| Outcome: | The proposed model outperforms two multilingual models on morphological probing, POS tagging and NER tasks in 9 languages. |
Learning Mutually Informed Representations for Characters and Subwords (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained language models rely on subword tokenization to process text as a sequence of subwords. |
| Approach: | They propose a character-subword language model that integrates character and subword modalities into one model. |
| Outcome: | The proposed model outperforms its backbone language models on English sequence labeling and classification tasks. |
MANTa: Efficient Gradient-Based Tokenization for End-to-End Robust Language Modeling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Subword tokenization algorithms have been an essential component of language modeling but their static nature results in important flaws that degrade the models’ downstream performance and robustness. |
| Approach: | They propose a module for Adaptive Neural TokenizAtion that is differentiable and trained end-to-end with the language model. |
| Outcome: | The proposed tokenizer improves robustness to character perturbations and out-of-domain data. |
Semantics or spelling? Probing contextual word embeddings with orthographic noise (2024.findings-acl)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) are used to generate contextual word embeddings . linguistics research has focused on semantic information in hidden states . |
| Approach: | They investigate whether a single character swap in the input word will not affect the resulting representation . they find that PLM-derived contextual word embeddings are highly sensitive to noise . |
| Outcome: | The results show that the PLM-derived representations are highly sensitive to noise . the fewer tokens used to represent a word at input, the more sensitive their CWE is . |
Kvistur 2.0: a BiLSTM Compound Splitter for Icelandic (2020.lrec-1)
Copied to clipboard
| Challenge: | Compounding is highly productive in Icelandic, and new compounds are constantly being created. |
| Approach: | They propose a character-based biLSTM model for splitting Icelandic compound words . the model learns how to split compound words into two parts . |
| Outcome: | The proposed model outperforms other methods on a corpus of manually split word forms. |
How Important Is Tokenization in French Medical Masked Language Models? (2024.lrec-main)
Copied to clipboard
| Challenge: | Word tokenization into subword units has become the prevailing standard in the field of natural language processing (NLP) over recent years . the precise factors contributing to its success remain unclear . |
| Approach: | They propose a tokenization strategy that integrates morpheme-enriched word segmentation into existing tokenization methods. |
| Outcome: | The proposed tokenization strategy outperforms character and word tokenization but the precise factors contributing to its success remain unclear. |
Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies using LLMs on psycholinguistic data have gone unverified . a growing body of research is using word-level prediction as a computational proxy . |
| Approach: | They compare morphological, morphologic, and BPE tokenization estimates with reading time data. |
| Outcome: | The proposed method could be used to evaluate morphological prediction. |
mALBERT: Is a Compact Multilingual BERT Model Still Worth It? (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies on the ethical and ecological impact of pre-trained language models raise questions about the temporal, financial, and environmental aspects of such models. |
| Approach: | They propose to focus on smaller models, such as compact models like ALBERT, which are more ecologically virtuous than these PLMs. |
| Outcome: | The proposed model is compared with classical multilingual models and is ethically virtuous. |
Exploring morphology-aware tokenization: A case study on Spanish language modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that subword tokenization improves performance of neural language models. |
| Approach: | They propose a linguistically grounded approach to train a tokenizer on morphologically segmented data. |
| Outcome: | The proposed tokenizer improves on a Spanish language model with morphological information. |
Corpus-Dependent Subcharacter Encoding via HMM-Guided Code Assignment (2026.acl-long)
Copied to clipboard
| Challenge: | Latom is a corpus-dependent alternative to byte encoding that learns fixed-length atomic codes from text. |
| Approach: | They propose a corpus-dependent alternative to byte encoding that learns fixed-length atomic codes from text. |
| Outcome: | The proposed framework improves text classification accuracy and reduces decoding errors. |